Statistical error in interpretation of aneurysm coil results.

Statistical Error in Interpretation of Aneurysm Coil Results To the Editor: We read with interest the recent report by Bendszus et al1 regarding the efficacy of new “bioactive” coils for cerebral aneurysm therapy. The authors note a “statistically significant” difference between outcomes in aneurysms treated with the Cerecyte coil and a matched pair cohort treated with bare platinum coils, with superior long-term occlusion rate for the Cerecyte group. We wish to call attention to a basic statistical error in their report, an error that lowered the calculated probability values to half of their true value, which might lead the reader to make an incorrect conclusion regarding the statistical superiority of Cerecyte coils. In the methods section, the authors state that 2-sided Fisher exact tests were performed “for comparison of the initial treatment results,” but they do not specify whether the outcome at 6 months was assessed by a 1-sided or a 2-sided test. We performed both 1-sided and 2-sided Fisher exact tests using the authors’ own data and found that they apparently are reporting the results of 1-sided tests. This use of a 1-sided test is inappropriate. The only situation in which a 1-sided test should be used is when a change in an unexpected direction would have absolutely no relevance to the study.2,3 This situation is unusual, and if there is any doubt regarding whether a 1or 2-sided test should be used, then a 2-sided test should be applied. The 1-sided test will yield a probability value that is half as small as the 2-sided test. This diminution of the probability value renders a 1-sided test very attractive to those whose definition of success is having a statistically significant result (ie, P 0.05). A 2-sided test is more conservative than a 1-tailed test because a 2-tailed test takes a more extreme test statistic to reject the null hypothesis. When deciding whether to use a 1or 2-sided statistic in the comparison between bare and bioactive coils, we submit that it remains entirely possible that Cerecyte could be associated with a higher recurrence rate than platinum coils. We are not saying that there is any evidence that Cerecyte causes a higher recurrence rate, but this is a realistic possibility, as demonstrated by results with the Matrix coil.4 As such, this possibility must be accounted for in the statistical evaluation of the data by using a 2-sided test. The 2-sided test gives us a probability value of 0.09. As a brief statistical refresher, the probability value indicates the probability of obtaining a result equal to or less consistent with the null hypothesis, if the null hypothesis were true. In the current case the null hypothesis is that there is no difference in recurrence rates between bare and Cerecyte coils. Thus, in the current series there is a 9% chance that a similar or more extreme angiographic outcome result, showing superiority of either platinum or Cerecyte, would be achieved even if the null hypothesis were true. Granted, this may indicate a trend toward superiority but not statistical superiority. Another statistical issue with this publication is the “power” values given in the results. It is not clear how these were calculated, but they were likely calculated retrospectively, which is of dubious value.5 Calculations of sample sizes for studies of new endovascular treatments for cerebral aneurysms indicate that the sample size of 55 patients in each group reported by Bendszus et al is much too small.6 Based on single-center retrospective data collected this far, we still have no idea if Cerecyte works in preventing aneurysm recurrences. And we won’t have any reasonable idea until we have completed a prospective, randomized controlled study with appropriate numbers of patients.